Facial recognition and monitoring device, system, and method

ABSTRACT

The present teachings relate to a device, system, and method for facial recognition and monitoring. The present teachings may relate to a method for recognizing and monitoring one or more users for an occurrence of a health event of the one or more users comprising: a) receiving incoming video data; b) preprocessing the video data; c) extracting facial data to identify one or more users; d) recognizing the presence, probability, and/or absence of the health event; e) generating one or more notifications or not generating based on the presence, probability, and/or absence of the health event; and f) optionally, enabling one or more safety protocols of a vehicle.

FIELD

The present teachings relate to a device, system, and method for facialrecognition and monitoring. The present teachings may find particularuse within vehicles. Using facial recognition and monitoring, thedevice, system, and method may be particularly useful in identifyingirregular health events associated with health conditions of a driverand/or passengers and initiating safety functions of a vehicle and/ormobile device upon the identification of the irregular health events.

BACKGROUND

Traffic accident injuries are the world's 8^(th) leading cause of death,killing approximately 1.35 million people each year, and affecting anadditional 20-50 million people with non-fatal injuries or disabilities.In the United States alone, approximately 6 million accidents occur eachyear, 90% of which can be attributed to incorrect driver and/or operatordecisions. As modern consumer technology continues to advance, driversand/or operators are becoming increasingly error-prone while driving,which may be partially attributed to the use of distracting, hand-helddevices. These driver and/or operator errors may be referred to asregular events which impact driving conditions.

In addition to traffic accidents related to driver and/or operatordecisions, the quick onset of an irregular health event related to amedical condition of a driver and/or passenger may result in unsafeoperating conditions. An irregular health event of a driver may quicklyimpact a driver's ability to safely operate a vehicle. An irregularhealth event of a passenger may quickly impact a driver's emotions andattention, also impacting a driver's ability to safely operate avehicle. An irregular health event may be associated with a medicalcondition of which an individual may have no control, no knowledge inadvance of the upcoming onset, or both. Some medical conditions, such asepilepsy, may be associated with frequent or infrequent irregular healthevents and may even limit the ability of an individual to obtain adriver's license, due to the uncertainty and likelihood of theoccurrence of the irregular health event.

What is needed is a device, system, and method which may be able todetect one or more regular events, irregular health events, or both.What is needed is a device, system, and method which may be able to beused within a vehicle. What is needed is a device, system, and methodwhich may be able to identify a regular event, an irregular healthevent, or both in one or more drivers, passengers, or both. What isneeded is a device, system, and method which may be able to provide adriver, with one or more medical conditions which may impact driving,the ability to drive a vehicle safely. What is needed is a device,system, and method which may be able to cooperate with a mobile device,in-vehicle communication system, or both to contact one or moreemergency services, pre-identified contacts, or a combination thereof.What is needed is a device, system, and method which may be able tocooperate with a driver assistance technology such that a vehicle isable to execute one or more emergency protocols. What is needed is adevice, system, and method which may be able to cooperate with a vehicleto reduce the likelihood of a traffic incident associated with a regularevent, irregular health event, or both.

SUMMARY

The present teachings relate to a method for detecting an occurrence ofan irregular health condition of one or more users comprising: a)receiving incoming video data by one or more processors related to theone or more users; b) preprocessing the video data by the one or moreprocessors into frame data, wherein the frame data includes one or moresingle frames, batches of frames, sequence of frames, or a combinationthereof; c) extracting facial data by the one or more processors fromthe frame data to identify the one or more users, wherein the facialdata includes one or more extracted faces, numeric array representationsof faces of the one or more users, one or more measurements and/orpredictions of one or more poses of the one or more users, or acombination thereof; d) determining the presence, probability, absence,or any combination thereof of the irregular health condition in the oneor more users by the one or more processors by comparing the facial datawith one or more stored facial data models accessible by the one or moreprocessors; e) generating one or more notifications based on recognizingthe presence and/or probability of the irregular health condition or notgenerating a notification based on the absence of the irregular healthcondition; and f) optionally, upon recognizing the presence and/orprobability of the irregular health condition, transmitting one or moreemergency signals from the one or more processors to a vehicle to enabledriver assistance technology to control driving of the vehicle in whichthe one or more users are located, such that the vehicle drives to andreaches a safe parking destination and/or turns on one or more emergencynotifiers of the vehicle.

The present teachings may provide for a system for facial recognitionand monitoring comprising i) one or more cameras; and ii) one or moreimage processing units in communication with the one or more cameras,including: a) one or more processors, b) one or more graphicsprocessors, c) one or more memory storage devices, and d) one or morenetwork connections. The one more image processing units and one or morecameras may be part of a recognition device.

The present teachings provide a device, system, and method which may beable to detect one or more health events. The health events may includeone or more regular events, irregular events or both. The presentingteachings provide a system which may be preprogrammed by one or morehumans, trained using one or more machine learning training models, orboth. The system may be provided with one or more video streams, images,and/or frames of a plurality of users experiencing one or more healthevents. Using facial recognition and machine learning, the system may beable to learn one or more facial recognition traits associated with oneor more health events. The present teachings provide a device, system,and method which may utilize a facial recognition and monitoring method(FRM method) to determine the presence and/or absence of one or morehealth events in a user. The FRM method of the present teachings may beuseful in identifying the health event. The present teachings mayprovide a recognition device which may be able to be integrated into avehicle. The present teachings may provide a system which is compatiblewith a mobile device of a user. Thus, whether the user is using arecognition device or mobile device, they user may be able to execute atleast a portion of FRM method while in a vehicle or any other setting.The present teachings provide a system which may be compatible with oneor more controls of the vehicle, mobile device, or both. The presentteachings may provide a system which is able to communicate with thevehicle, mobile device, or both and execute one or more safetyprotocols. The safety protocols may allow for a vehicle to initiatevehicle assistance technology to maneuver a vehicle into a saferposition. The safety protocols may include initiating contact withemergency services, pre-identified contacts, or both to alert and/orsend for assistance for the user if they are experiencing a healthevent.

The present teachings may provide an unconventional approach atrecognizing and monitoring users via facial recognition, as not onlydoes the system monitor a user's behavior, but also their healthconditions. Furthermore, typical systems may function to alert a driverof a conscious behavior while driving so as to refocus the attention ofthe driver to the road and surrounding driving conditions to adequatelycontrol the vehicle. The present teachings provide a device, system, andmethod for not only identifying irregular health events which may beunpredicted and not under the control of the user, but even compensatingwhen a driver is unable to maintain control of a vehicle due to theseverity of a health event.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 illustrates a recognition device integrated into a vehicleaccording to the teachings herein.

FIG. 2A illustrates a recognition device according to the teachingsherein.

FIG. 2B illustrates a recognition device according to the teachingsherein.

FIG. 3 illustrates a hardware configuration of a recognition deviceaccording to the teachings herein.

FIG. 4 illustrates a facial recognition and monitoring method executedby a recognition device according to the teachings herein.

FIG. 5 illustrates a schematic of a facial recognition and monitoringsystem according to the teachings herein.

FIG. 6 illustrates the network architecture of the cloud-based networkaccording to the teachings herein.

FIG. 7 illustrates a development module according to the teachingsherein.

FIG. 8 illustrates a remotely executed facial recognition and monitoringmethod executed by a cloud-based network according to the teachingsherein.

FIG. 9 illustrates a method for facial recognition and monitoring forthe presence of a health event according to the teachings herein.

FIG. 10 illustrates an advanced driver assistance system integrated witha vehicle according to the teachings herein.

FIG. 11 illustrates a model testing and training method according to theteachings herein.

DETAILED DESCRIPTION

The explanations and illustrations presented herein are intended toacquaint others skilled in the art with the present teachings, itsprinciples, and its practical application. The specific embodiments ofthe present teachings as set forth are not intended as being exhaustiveor limiting of the present teachings. The scope of the present teachingsshould be determined with reference to the appended claims, along withthe full scope of equivalents to which such claims are entitled. Thedisclosures of all articles and references, including patentapplications and publications, are incorporated by reference for allpurposes. Other combinations are also possible as will be gleaned fromthe following claims, which are also hereby incorporated by referenceinto this written description.

Setting(s) for Facial Recognition and Monitoring System and Method

The facial recognition and monitoring system and method of the presentteachings may be found useful in multiple settings. Settings may includewithin one or more vehicles, residential settings (e.g., homes),education settings (e.g., classrooms), medical settings (e.g.,hospitals), the like, or a combination thereof. The system and methodmay be advantageous for use within vehicles to not only identify thepresence of one or more human conditions, but also activating one ormore emergency features of the vehicle to maintain safety of the driver,passengers, and other vehicles and individuals in proximity of thevehicle. One or more vehicles may include one or more private passengervehicles, public passenger vehicles, or both. One or more privatepassenger vehicles may include one or more automobiles, boats, bicycles,motorcycles, wheelchairs, small passenger planes, helicopters,recreational vehicles (e.g., RV, ATV, OTV), the like, or a combinationthereof. One or more automobiles may include one or more cars, trucks,sports utility vehicles, cross-over vehicles, the like, or anycombination thereof. One or more public passenger vehicles may includeone or more buses, trains, boats, planes, chairlifts, the like, or anycombination thereof. The one or more settings may be suitable for atleast temporarily accommodating one or more users.

The facial recognition and monitoring system and method of the presentteachings may be employed by one or more users. One or more users mayinclude a single user or a plurality of users. One or more users may bewithin the same setting or different settings. One or more users mayinclude one or more humans, animals, or both. One or more animals mayinclude one or more domestic pets, wild animals, the like, or anycombination thereof. The one or more users may include one or moredrivers, passengers, or both of one or more vehicles. The one or moreusers may have one or more known health conditions or may be free or oneor more health conditions. The one or more users may be aware and/orunaware of the presence of one or more health conditions.

Health Condition and Health Event

The facial recognition and monitoring system and method of the presentteachings may find particular use in identifying the occurrence of oneor more health events. The one or more health events may include one ormore health events that may make driving and/or being a passenger in avehicle at least temporarily unsafe for one or more passengers, drivers,or both of the same vehicle, other vehicles in proximity, otherindividuals in proximity to the user (e.g., pedestrian), the like, orany combination thereof. The one or more health events may include oneor more events in which the ability to focus on driving, to physicallycontrol driving of the vehicle, or both may be at least temporarilyimpaired. The one or more health events may include one or moreirregular health events, regular events, or both.

One or more health events may include one or more irregular healthevents. One or more irregular health events may be tied to one or moremedical conditions, behaviors, or both of one or more users. One or moreirregular health events may be unable to be controlled by a user oncethe event is occurring. One or more irregular health events may includeone or more health conditions with a quick onset, slow onset, no knownsymptoms in advance, spontaneous occurrence, the like, or anycombination thereof. One or more irregular health events may be a resultof a behavior of one or more users, may be known to result by one ormore users, or both. One or more irregular health events may be one ormore health events of which a user (e.g., individual) may have nocontrol, knowledge of an upcoming onset, or both. One or more irregularhealth events may include one or more seizures, heart attacks, strokes,fainting, falling asleep, drowsiness, inebriation, vomiting, the like,or any combination thereof. One or more health conditions may includeepilepsy, high cholesterol, high blood pressure, aneurysm, pregnancy,narcolepsy, dehydration, the like, or any combination thereof. One ormore behaviors may include alcohol consumption, drug consumption, lackof sleep, the like, or any combination thereof.

In addition to irregular events, regular events may be experienced by auser which may make the driving experience unsafe. Regular events may bethose which may be caused by a user, the user is consciously and/orsubconsciously aware of, the user may consciously choose to engage in,the like, or any combination thereof. Regular events may includeheightened emotions (e.g., anger toward a fellow passenger, sadness at apersonal situation etc.), attention or lack thereof to the road anddriving conditions (e.g., engaging in deep conversation with a fellowpassenger), looking away from the road, driving habits (e.g., tendencyto speed or tailgate while angry or stressed), texting and/or otherwiselooking at a mobile device, the like, or any combination thereof.

The one or more users may be recognized and monitored by a recognitiondevice.

Recognition Device

The present teachings may relate to a recognition device. Therecognition device may be useful in recognizing and monitoring one ormore users, such as in one or more settings. The recognition device maybe useful for locating within one or more settings. The recognitiondevice may be temporarily, semi-permanently, and/or permanently affixedand/or located within a setting. For example, a recognition device maybe permanently affixed within a vehicle. As another example, arecognition device may be temporarily or semi-permanently affixed withina vehicle. A recognition device may be located within any portion of avehicle in which one or more cameras may have line of sight on one ormore users within the vehicle. The one or more users may include one ormore drivers, passengers, or both. The recognition device may be atleast partially located on a rearview mirror, headliner, steering wheel,visor, instrument panel, entertainment center, internal vehicle control,passenger seats, headrests, panels (e.g., doors) within a vehicle, doortrim, pillars (e.g., A, B, and/or C-pillar), consoles (e.g., centerconsole), the like, or any combination thereof. The recognition devicemay include one or more housings. A housing may function to retain oneor more components of the recognition device. A housing may bepermanently, semi-permanently, or not affixed to one or more portions ofa vehicle or within other settings. For example, a housing may bepermanently located within an instrument panel of a vehicle. As anotherexample, a housing may be temporarily attached to any interior portionof a vehicle (e.g., mounting hardware, suction cups, hooks, the like, orany combination thereof. As another option, a housing may rest within aholder of a vehicle. For example, a cup holder, tray, and the likewithin the vehicle. The recognition device may include one or moresensing devices (e.g., cameras), sensing device connections, sensingdevice modules, image processing units, processors, memory storagedevices, housings, applications, network connections, power supplies,the like, or any combination thereof.

The recognition device may include one or more viewer sensing devices.The one or more sensing devices may function to detect the presence of auser, recognize a user, monitor a user, the like, receive and/ortransmit data related to the user, or any combination thereof. The oneor more sensing devices may include one or more cameras, motion sensors,heart rate monitors, breathing monitors, the like, or any combinationthereof. One or more sensing devices may include one or more cameras(e.g., camera modules). The one or more cameras may be suitable forcapturing one or more videos, images, frames, the like, or anycombination thereof. The one or more cameras may be positioned within asetting to have a line of sight on one or more users. Line of sight maymean the camera is in view of at least part of or all of a person'sface, at least part of or all of a person's body (e.g., upper torso,arms, shoulders, etc.), or any combination thereof. Line of sight maymean having the user's eyes, nose, mouth, ears, neck, or any combinationthereof in view of the camera. The one or more cameras may have a lineof sight (e.g., have in view) a single user or a plurality of users. Forexample, a camera may be positioned in a vehicle to have a line of sighton a face of a driver. As another example, a camera may be positioned ina vehicle to have a line of sight on a face of a passenger in the frontpassenger seat. As a further example, a camera may be positioned in avehicle to have a line of sight on the faces of passengers in rearpassenger seats of a vehicle. The camera may point toward a rear, side,and/or front of a vehicle. Pointing generally toward a rear of a vehiclewill provide for one or more front-facing users to have their face(s)within the line of sight of the camera. The recognition device may befree of one or more sensing devices and be in communication with one ormore sensing devices. The one or more cameras may have a wide-angle lens(e.g., viewing angle of 150 degrees or greater). The one or more camerasmay be capable of capturing static images, video recordings, or both atresolutions of about 480 pixels or greater, 640 pixels or greater, 720pixels or greater, or even 1080 pixels or greater. The one or morecameras may be able to capture video recordings at a frame rate of about25 frames per second or greater, about 30 frames per second or greater,about 60 frames per second or greater, or even 90 frames per second orgreater. A suitable camera for use with the recognition device mayinclude the SainSmart IMX219 Camera Module with an 8MP sensor and160-degree field of vision, the camera module and its specificationsincorporated herein by reference for all purposes.

The recognition device may include or be connectable with one or moresensing device connections. The one or more sensing device connectionsmay function to connect one or more sensing devices to the recognitiondevice, an image processing unit, a power supply, the like, or anycombination thereof. The one or more sensing device connections may bewired, wireless, or both. The one or more sensing device connections mayinclude one or more communication wires connecting one or more sensingdevices to one or more image processing units. For example, the one ormore sensing device connections may include a wire connecting a camera(e.g., camera module) to an image processing unit. The one or moresensing device connections may be any type of cable and or wire suitablefor transferring data including video, images, frames, sound, the like,or any combination thereof. The one or more sensing device connectionsmay allow for one or more sensing devices to be located within apassenger area of a vehicle while the image processing unit is locatedwithin a controls section of a vehicle such that they are separate fromone another. The one or more sensing device connections may allow forone or more sensing devices to be located within a same housing as animage processing unit.

The recognition device may include one or more image processing units.One or more image processing units may function to receive, process,transmit image data, or any combination thereof; store image data; orboth. The one or more image processing units may include one or moreprocessors, memory storage devices, circuit boards, the like, or anycombination thereof. The one or more image processing units may includeone or more central processing units, graphics processing units, memorymediums, storage mediums, the like, or any combination thereof. The oneor more image processing units may include an electronic circuit boardor similar. The electronic circuit board may house and place one or moreprocessing units and memory storage devices in communication with oneanother. The electronic circuit board may be in communication with oneor more sensing device modules, power supply sources, networkconnections, the like, or any combination thereof. The electroniccircuit board may be located within a housing of a recognition device.The electronic circuit board may be housed in a same or differenthousing as one or more sensing devices.

The one or more image processing units may include one or moreprocessors. One or more processors may function to analyze image data,execute instructions, transmit image data, or any combination thereof.The one or more processors may be located within the recognition device.The one or more processors may be located within a same or separatehousing as one or more sensing devices. The one or more processors mayinclude a single or a plurality of processors. The one or moreprocessors may function to process data, execute one or moreinstructions to analyze data, or both. Processing data may includereceiving, transforming, outputting, executing, the like, or anycombination thereof. One or more processors may be in communication withone or more memory storage devices. One or more processors may accessand execute one or more instructions stored within one or more memorymediums. One or more processors may store processed data within one ormore storage mediums. One or more processors may be part of one or morehardware, software, systems, or any combination thereof. The one or moreprocessors may be referred to as one or more electronic processors. Oneor more hardware processors may include one or more central processingunits, multi-core processors, front-end processors, graphics processors,the like, or any combination thereof. One or more processors may includeone or more central processing units (CPU), graphics processing units(GPU), or both. One or more processors may be in communication with,work together with, or both one or more other processors. For example, acentral processing unit may cooperate with a graphics processing unit.One or more processors may be a processor, microprocessor, electroniccircuit, the like, or a combination thereof. For example, a centralprocessing unit may be a processor or microprocessor. A centralprocessing unit may function to execute instructions stored within amemory medium of the recognition device. An exemplary central processingunit may include the Cortex-A57 processor provided by Arm Limited, theprocessor and its specifications incorporated herein by reference forall purposes. As an example, a graphics processing unit may include oneor more electronic circuits. A graphics processing unit may function toaccelerate creation and rendering of image data. Image data may includeimages, videos, animation, frames, the like, or a combination thereof.The graphics processing unit may be beneficial in performing fast mathcalculations associated with the image data and freeing up processingcapacity of the central processing unit. An exemplary graphicsprocessing unit may include the GeForce® GTX 1650 D6 0C Low Profile 4G(model number GV-N1656OC-4GL) by GIGA-BYTE Technology Co., the moduleand its specifications incorporated herein by reference. The one or moreprocessors may be non-transient. The one or more processors may convertincoming data to data entries to be saved within one or more storagemediums.

One or more image processing units may include one or more memorystorage devices (e.g., electronic memory storage device). The one ormore memory storage devices may function to store data, databases,instructions, or any combination thereof. The one or more memory storagedevices may include one or more hard drives (e.g., hard drive memory),chips (e.g., Random Access Memory “RAM)”), discs, flash drives, memorycards, the like, or any combination thereof. One or more discs mayinclude one or more floppy diskettes, hard disk drives, optical datastorage media including CD ROMs, DVDs, and the like. One or more chipsmay include ROMs, flash RAM, EPROMs, hardwired or preprogrammed chips,nanotechnology memory, or the like. The data stored within one or morememory storage devices may be compressed, encrypted, or both. The one ormore memory storage devices may be located within a recognition device.One or more memory storage devices may be non-transient. One or morememory storage devices may include one or more memory mediums, storagemediums, the like, or a combination thereof. One or more memory mediumsmay store one or more computer executable instructions. One or morememory mediums may store one or more algorithms, methods, rules, thelike, or any combination thereof. One or more memory mediums may beaccessible by one or more processors (e.g., CPU, GPU) to read and/orexecute one or more computer executable instructions. An exemplarymemory medium may include the Kingston ValueRAM 4 GB DDR SDRAM MemoryModule, the memory module and its specifications incorporated herein byreference. One or more storage mediums may store one or more databases.One or more storage mediums may store one or more algorithms, methods,rules, the like, or a combination thereof. One or more storage mediumsmay store one or more computer executable instructions. Instructions,algorithms, methods, rules, and the like may be transferred from one ormore storage mediums to one or more memory mediums for accessing andexecution by one or more processors. One or more storage mediums maystore one or more data entries in a native format, foreign format, orboth. One or more storage mediums may store data entries as objects,files, blocks, or a combination thereof. One or more storage mediums maystore data in the form of one or more databases. One or more storagemediums may store data received from one or more processors (e.g., CPU,GPU). One or more storage mediums may store data received from one ormore processors after the one or more processors execute one or moreinstructions from one or more memory mediums. An exemplary storagemedium may include the 16 GN eMMC Module XU4 Linux sold by HardKernelCo., Ltd., the module and its specifications incorporated herein byreference.

The recognition device may include one or more power supplies. A powersupply may function to provide electric power to an electrical load of arecognition device, transmit power to an image processing unit, or both.A power supply may be any device capable of converting electric currentfrom a power source to a correct voltage, current, and/or frequency topower the electrical load of the recognition device. A power supply maybe located in the same or a different housing as an image processingunit. A power supply may have one or more power input connections, poweroutput connections, or both. A power input connection may function toconnect to a power input, receive energy in the form of electric currentfrom a power source, or both. A power source may include an electricaloutlet, energy storage devices, or other power supplies. A power sourcemay be a power source of a vehicle. A power source may include one ormore batteries, fuel cells, generators, alternators, solar powerconverters, the like, or any combination thereof. A power outputconnection may function to connect to one or more electric components ofa recognition device, connect to an image processing unit, deliverycurrent to one or more electrical loads of the recognition device, orany combination thereof. The power input connection, power outputconnection, or both may be wired (e.g., hardwired circuit connection) orwireless (e.g., wireless energy transfer). An exemplary power supply maybe the RS-15-24 single output switching power supply by MEAN WELL whichis an AC/DC 15 W 24V single output power supply, the power supply andits specifications incorporated herein by reference.

The recognition device may include one or more network connections. Theone or more network connections may function to place the recognitiondevice in communication with one or more networks. The one or morenetwork connections may be in communication with and/or connected to oneor more image processing units. The one or more network connections maybe in communication with one or more networks. The one or more networkconnections may be suitable for connecting to the Internet. The one ormore network connections may include one or more IoT (i.e., “Internet ofThings) connections. The one or more network connections may be wired,wireless, or both. An exemplary network connection may include the INP1010/1011 multi-protocol wireless module provided by InnoPhase, themodule and its specifications incorporated herein by reference.

The recognition device may include one or more application layers. Theone or more application layers may function to retain one or more facialrecognition and monitoring methods, provide accessible computerexecutable instructions, be accessible by one or more image processingunits, the like, or any combination thereof. The one or more applicationlayers may be software (e.g., computer executive instructions). The oneor more application layers may be in communication with one or morenetworks. The one or more application layers may be able to be created,updated, or otherwise modified remotely via one or more networks and oneor more network connections.

Facial Recognition and Monitoring System (“FRM System”)

The FRM system may include one or more physical data centers. A physicaldata center may function to house one or more components of the FRMsystem. The physical data center may function to host one or moreservers, processors, memory storage devices, networks, the like, or anycombination thereof. The physical data center may function to provide anon-transient host location for one or more components of the FRMsystem. The physical data center may include at about 1 terabyte ofstorage space or greater, about 2 terabytes of storage space or greater,or even 5 terabytes of storage space or greater. The physical datacenter may operate one or more components on one or more operatingsystems. One or more operating systems may include Linux, Windows,MacOS, ArcaOS, Haiku, ReactOS, FreeDOS, Wayne OS, the like, or anycombination thereof. The physical data center may function to host oneor more modules, execute one or more modules, or both of a cloud-basednetwork.

The system may include one or more processors. The one or moreprocessors may function to analyze one or more data from one or moresensing devices, memory storage devices, databases, user interfaces,recognition devices, modules, the like or any combination thereof;convert one or more incoming data signals to data suitable for analysisand/or saving within a database (e.g., data conversion, data cleaning);or a combination thereof. One or more processors may be included in oneor more user interfaces, servers, computing devices, the like, or anycombination thereof. The one or more processors may or may not becloud-based (e.g., remote from other portions of the system). One ormore processors hosted by a physical data center may be consideredcloud-based. One or more processors hosted remotely from one or morerecognition devices, personal computing devices, or both may beconsidered cloud-based. One or more processors may include a single or aplurality of processors. One or more processors may be in communicationwith one or more other processors. One or more processors may beassociated with and/or execute one or more modules of one or moresystems. The one or more processors may function to process data,execute one or more algorithms to analyze data, or both. Processing datamay include receiving, transforming, outputting, executing, the like, orany combination thereof. One or more processors may be part of one ormore hardware, software, systems, or any combination thereof. One ormore hardware processors may include one or more central processingunits, multi-core processors, front-end processors, graphics processingunits, the like, or any combination thereof. The one or more processorsmay be non-transient. The one or more processors may be referred to asone or more electronic processors. The one or more processors mayconvert data signals to data entries to be saved within one or morememory storage devices. The one or more processors may access one ormore algorithms (e.g., computer executable instructions) saved withinone or more memory storage mediums.

The system may include one or more memory storage devices (e.g.,electronic memory storage device). The one or more memory storagedevices may store data, databases, algorithms, processes, methods, orany combination thereof. The one or more memory storage devices mayinclude one or more hard drives (e.g., hard drive memory), chips (e.g.,Random Access Memory “RAM”), discs, flash drives, memory cards, thelike, or any combination thereof. One or more discs may include one ormore floppy diskettes, hard disk drives, optical data storage mediaincluding CD ROMs, DVDs, and the like. One or more chips may includeROMs, flash RAM, EPROMs, hardwired or preprogrammed chips,nanotechnology memory, or the like. The one or more memory storagedevices may include one or more cloud-based storage devices. One or morememory storage devices located remote from one or more recognitiondevices, personal computing devices, and/or user interfaces, may beconsidered a cloud-based storage device. The data stored within one ormore memory storage devices may be compressed, encrypted, or both. Theone or more memory storage devices may be located within one or morecomputing devices, servers, processors, user interfaces, or anycombination thereof. One or more memory storage devices may be referredto as one or more electronic memory storage devices. One or more memorystorage devices may be non-transient. One or more memory storage mediumsmay store one or more data entries in a native format, foreign format,or both. One or more memory storage mediums may store data entries asobjects, files, blocks, or a combination thereof. The one or more memorystorage mediums may include one or more algorithms, methods, rules,databases, data entries, the like, or any combination thereof storedtherein. The one or more memory storage mediums may store data in theform of one or more databases.

One or more computing devices may include one or more databases. The oneor more databases may function to receive, store, and/or allow forretrieval of one or more data entries. Data entries may be one or morevideo streams, frames, identifiers, frame analysis results, user data,and the like associated with one or more users. The one or moredatabases may be located within one or more memory storage devices. Theone or more databases may include any type of database able to storedigital information. The digital information may be stored within one ormore databases in any suitable form using any suitable databasemanagement system (DBMS). Exemplary storage forms include relationaldatabases (e.g., SQL database, row-oriented, column-oriented),non-relational databases (e.g., NoSQL database), correlation databases,ordered/unordered flat files, structured files, the like, or anycombination thereof. The one or more databases may store one or moreclassifications of data models. The one or more classifications mayinclude column (e.g., wide column), document, key-value (e.g., key-valuecache, key-value store), object, graph, multi-model, or any combinationthereof. One or more databases may be located within or be part ofhardware, software, or both. One or more databases may be stored on asame or different hardware and/or software as one or more otherdatabases. One or more databases may be located in a same or differentnon-transient storage medium as one or more other databases. The one ormore databases may be accessible by one or more processors to retrievedata entries for analysis via one or more algorithms, methods, rules,processes, or any combination thereof. The one or more databases mayinclude a single database or a plurality of databases. One database maybe in communication with one or more other databases. One or more otherdatabases may be part of or separate from the system. One or moredatabases may be connected to one or more other databases via one ormore networks. Connection may be wired, wireless, the like, or acombination thereof. For example, a database of the system may be incommunication with one or more other databases via the Internet. Thedatabase may also receive one or more outputs of the system. Thedatabase may be able to have the data outputted, sorted, filtered,analyzed, the like, or any combination thereof. The database may besuitable for storing a plurality of records.

The system may include one or more applications. The application (i.e.,“computer program”) may function to access data, upload data, or both tothe FRM system, interaction with a user interface, or any combinationthereof. The application may be stored on one or more memory storagedevices. The application may be stored on one or more personal computingdevices. The application may comprise and/or access one or morecomputer-executable instructions, algorithms, rules, processes, methods,user interfaces, menus, databases, the like, or any combination thereof.The computer-executable instructions, when executed by a computingdevice may cause the computing device to perform one or more methodsdescribed herein. The application may be downloaded, accessible withoutdownloading, or both. The application may be downloadable onto one ormore computing devices. The application may be downloadable from anapplication store (i.e., “app store”). An application store may include,but is not limited to, Apple App Store, Google Play, Amazon Appstore, orany combination thereof. The applicable may be accessible withoutdownloading onto one or more computing devices. The application may beaccessible via one or more web browsers. The application may beaccessible as a website. The application may interact and/orcommunication through one or more user interfaces. The application maybe utilized by one or more computing devices. The application may beutilized on one or more computing devices. The application may also bereferred to as a dedicated application.

One or more computing devices (e.g., personal computing devices) mayinclude one or more user interfaces. The one or more user interfaces mayfunction to display information related to a user, receive user inputsrelated to a user, display data and/or one or more prompts to a user, orany combination thereof. The one or more user interfaces may be suitablefor receiving data from a user. The one or more user interfaces mayinclude one or more graphic user interfaces (GUI), audio interfaces,image interfaces, the like, or any combination thereof. One or moregraphic user interfaces may function to display visual data to a user,receive one or more inputs from the user, or both. The one or moregraphic interfaces may include one or more screens. The one or morescreens may be a screen located on a computing device. The one or morescreens may be a screen on a mobile computing device, non-mobilecomputing device, or both. The one or more graphic interfaces mayinclude and/or be in communication with one or more user input devices,audio interfaces, image interfaces, the like, or any combinationthereof. The one or more user input devices may allow for receiving oneor more inputs from a user. The one or more input devices may includeone or more buttons, wheels, keyboards, switches, mice, joysticks, touchpads (i.e., a touch-sensitive area, provided as a separate peripheral orintegrated into a computing device, that does not display visualoutput), touch-sensitive monitor screens, microphones, the like, or anycombination thereof. The one or more input devices may be integratedwith a graphic user interface. An audio interface may function toproject sound to a user and/or receive sound from a user. The audiointerface may include audio circuitry, one or more speakers, one or moremicrophones, the like, or any combination thereof. An image interfacemay function to capture, receive, display, and/or transmit one or moreimages. An image interface may include one or more cameras. A userinterface may function to display and/or navigate through one or moremenus of the application.

The system may include one or more computing devices. The one or morecomputing devices may function to allow a user to interact with anapplication, the system, or both; execute one or more algorithms,methods, and/or processes; receive and/or transmit one or more signals,convert one or more signals to data entries, retrieve one or more dataentries from one or more storage mediums, or any combination thereof.The one or more computing devices may include and/or be in communicationwith one or more processors, memory storage devices, servers, networks,user interfaces, recognition devices, other computing devices, the like,or any combination thereof. The one or more or more computing devicesmay be in communication via one or more interaction interfaces (e.g., anapplication programming interface (“API”)). The computing device may beone or more personal computers (e.g., laptop or desktop), mobile devices(e.g., mobile phone, tablet, smart watch, etc.), or any combinationthereof. The computing device may include one or more personal computingdevices. Personal computing devices may be computing devices typicallyused be a single person, having a log-in or sign-in function or otheruser authentication, can store and relay information privately to auser, the like, or a combination thereof. Personal computing devices mayhave the ability to transmit one or more notifications to one or moreemergency services, predetermined contacts, the like, or a combinationthereof. Personal computing devices may have the ability to send and/orreceive one or more text messages, SMS messages, push notifications,emails, phone calls, the like, or any combination thereof.

The system of the present disclosure may be integrated and/or includeone or more networks. The physical data center, one or more userinterfaces, one or more personal computing devices, one or morerecognition devices, one or more cloud networks, or any combinationthereof may be in selective communication with one or more networks. Theone or more networks may be formed by placing two or more computingdevices in communication with one another. One or more networks mayinclude one or more physical data centers, communication hubs,communication modules, computing devices, processors, databases,servers, memory storage devices, recognition devices, sensing devices,the like, or any combination thereof. One or more networks may be freeof and/or include one or more communication hubs (e.g., router, wirelessrouter). One or more components of the system may be directly connectedto one another without the use of a communication hub. One or morenetworks may be connected to one or more other networks. One or morenetworks may include one or more local area networks (“LAN”), wide areanetworks (“WAN”), virtual private network (“VPN”), intranet, Internet,cellular networks, the like, or any combination thereof. The network maybe temporarily, semi-permanently, or permanently connected to one ormore computing devices, recognition devices, user interfaces, the like,or any combination thereof. A network may allow for one or morecomputing devices, recognition devices, user interfaces, physical datacenters, or a combination thereof to be connected to one or moreportions of the system, transmit data between one or more components ofthe system, or any combination thereof. One or more portions of thenetwork hosted at one or more physical data centers may form one or morecloud-based network, recognition device management networks, videostream management network, the like, or any combination thereof.

The system may include one or more recognition device managementnetworks. The one or more recognition device management networks mayfunction to receive and collect data from one or more recognitiondevices. The one or more recognition device management networks may beat least partially hosted by the same or different physical data centersas the cloud-based network, video stream management network, or both.The one or more recognition device management networks may include oneor more remote storage devices, recognition devices, processors, or acombination thereof. The one or more remote storage devices may includeone or more main storage devices, node storage devices, or both. One ormore node storage devices may be layered between and in communicationwith both a plurality of recognition devices and a main storage device.One or more node storage devices may function to predetermined data froma plurality of recognition devices. Each node storage device may beassociated with a different set of predetermined data from the same ordifferent recognition devices as another node storage device. The nodestorage devices may organize collected data into groups. For example,one node storage device may collect data for users in publictransportation vehicles. As another example, one node storage device maycollect data associated with users in private passenger vehicles. As afurther example, one node storage device may collect data for users ofages 20-29 while another collected data for users ages 30-39 (e.g.,nodes assigned by age of users). The one or more node storage devicesmay append one or more identifiers to collected data (e.g., uniqueidentifiers), collect all data output received from a recognition deviceor both. The one or more node storage devices may transmit collecteddata to one or more main storage devices.

The system may include one or more video stream management networks. Theone or more video stream management networks may function to receive andcollect data from one or more personal computing devices, web servers,user interfaces, the like, or any combination thereof. The one or morevideo stream management networks may be at least partially hosted by thesame or different physical data centers as the cloud-based network,recognition device management network, or both. The one or more videostream management networks may include one or more remote storagedevices, processors, personal computing devices, web servers, or anycombination thereof. The one or more video stream management networksmay include one or more video stream clouds. One or more video streamclouds may include one or more memory storage devices, processors, webservers, the like, or a combination thereof. One or more video streamclouds may include video data stored therein. Video data may include oneor more video streams, image data, frames, the like, or a combinationthereof stored therein. One or more video stream clouds may associateone or more uniform resource locators (URL) to one or more video datafiles. Each video data file stored within a video stream cloud may beassociated with its own URL. The one or more video stream clouds may beprovided by a video streaming and hosting services within or remotelyaccessible to the cloud-based network, one or more personal computingdevices, or both. The one or more video stream clouds may be configuredto accept video streams using Real Time Streaming Protocol (RTSP),Internet Protocol (IP), the like, or a combination thereof. The videostreams may be received by the video stream cloud from one or more uservideo clouds, user data centers, or both. A video stream cloud may bevideo cloud storage (e.g., remote storage) for videos of a specificuser. For example, a user may have their own user authentication to login to and save video stream files into their video stream cloud. A userdata center may be a computing device of a user which is able to storevideo data internally.

The one or more remote storage devices may include one or more mainstorage devices, node storage devices, or both. One or more node storagedevices may be layered between and in communication with both aplurality of recognition devices and a main storage device. One or morenode storage devices may function to predetermined data from a pluralityof recognition devices. Each node storage device may be associated witha different set of predetermined data from the same or differentrecognition devices as another node storage device. The node storagedevices may organize collected data into groups. For example, one nodestorage device may collect data for users in public transportationvehicles. As another example, one node storage device may collect dataassociated with users in private passenger vehicles. As a furtherexample, one node storage device may collect data for users of ages20-29 while another collected data for users ages 30-39 (e.g., nodesassigned by age of users). The one or more node storage devices mayappend one or more identifiers to collected data (e.g., uniqueidentifiers), collect all data output received from a recognition deviceor both. The one or more node storage devices may transmit collecteddata to one or more main storage devices.

A cloud-based network may include one or more device and data managementmodules, cloud computing and data management modules, features andanalytics modules, data storage modules, execution data managementmodules, development modules, the like, or any combination thereof.

The system may include a device and data management module. The deviceand data management module may function to collect, organize, and storedata received from one or more recognition devices. Data may includevideo streams, frames, images, sound, frame analysis results, user data,recognition device data, the like, or any combination thereof. Thedevice and data management module may include recognition device eventmanagement, recognition device management, and recognition devicestorage. The device data management module may be in communication,either directly and/or indirectly, with one or more recognition devices.The device data management module may be in communication with one ormore remote storage devices. The device data management module collectdata from one or more recognition device via recognition devicemanagement. Recognition device management may include one or more IoTgateways, connections, and the like. Recognition device management maybe any hardware and/or software component suitable for receiving datafrom one or more recognition devices, remote storage devices, or both.The collected data is then collected and organized by recognition deviceevent management. The recognition device event management may includeone or more processors. Recognition device event management may filterand sort collected data from one or more recognition devices.Recognition device event management may sort collected data by users,timestamps, ages of users, genders of users, health events found,geolocations, the like, or any combination thereof. Recognition devicemanagement may associate data related to one or more users to one ormore user records. Recognition device management may transmit collectedand organized data to recognition device storage. Recognition devicestorage may include one or more memory storage devices. Recognitiondevice storage may include one or more memory mediums, storage mediums,or both. Recognition device storage may include one or more databases.Recognition devices storage may provide for long-term storage,short-term storage, or both. The recognition device storage may transmitcollected data to one or more data storage modules. The recognitiondevice storage may transmit collected data to execution storage forlong-term storage.

The system may include a cloud computing and data management module. Thecloud computing and data management module may function to collect,organize, and store data received from one or more personal computingdevices, web servers, the like, or a combination thereof. Data mayinclude video streams, frames, images, sound, frame analysis results,user data, recognition device data, the like, or any combinationthereof. The cloud computing and data management module may include acloud computing module, cloud event management, cloud computing storage,the like, or a combination thereof. The cloud computing module may be incommunication, either directly and/or indirectly, with one or morepersonal computing devise, web servers, the like, or a combinationthereof. The cloud computing module may be in communication with one ormore remote storage devices. The cloud computing and data managementmodule may collect data from one or more personal computing devices, webservers, remote storage devices, or any combination thereof via thecloud computing module. The cloud computing module may include one ormore IoT gateways, connections, and the like. The cloud computing modulemay include one or more network/IP/S3/RSTP gateways. The cloud computingmodule may be any hardware and/or software component suitable forreceiving data from one or more personal computing devise, web servers,remote storage devices, or both. The collected data is then collectedand organized by cloud computing. Cloud computing may include one ormore processors. Cloud computing may filter and sort collected data fromone or more personal computing devices, web servers, remote storagelocations, or a combination thereof. Cloud computing may sort collecteddata by users, timestamps, ages of users, genders of users, healthevents found, geolocations, the like, or any combination thereof. Cloudcomputing may associate data related to one or more users to one or moreuser records. Cloud computing may transmit collected and organized datato cloud computing storage. Cloud computing storage may include one ormore memory storage devices. Cloud computing storage may include one ormore memory mediums, storage mediums, or both. Cloud computing storagemay include one or more databases. Cloud computing storage may providefor long-term storage, short-term storage, or both. Cloud computingstorage may transmit collected data to one or more data storage modules.The cloud computing storage may transmit collected data to executionstorage for long-term storage.

The system may include one or more data storage modules. The one or moredata storage modules may function to store received data for short-termstorage, long-term storage, further analytics, the like, or acombination thereof. One or more data storage modules may include asingle or a plurality of storage modules. One or more data storagemodules may include execution event storage, cloud storage, executionstorage, the like, or a combination thereof. One or more data storagemodules may be located within a cloud-based network. One or more datastorage modules may include one or more servers, memory storage devise,the like, or a combination thereof. One or more data storage module mayinclude one or more databases stored therein. Execution event storagemay provide for an organized record of all detected health events, aportion of detected health events, or both. Organized may mean inchronological sequence from most recent detected event to oldestdetected event. A portion of detected health events may be over apredetermined period of time. A period of time may be a predeterminedtime frame from the present day and back. For example, 6 months, 1 year,5 years, or even 10 years. Execution storage may provide for long-termstorage of data received from recognition device storage, cloudcomputing storage, or both. Execution storage may be accessible by oneor more features and analytics modules. Execution storage may collectdata from one or more device and data management modules, cloudcomputing and data management modules, or both. Execution storage maycollect data which includes organized collections of labeled frames,labeled video sequences, extracted data in table format, extracted datain data frames format, other output data from one or more recognitiondevices, the like, or any combination thereof. Cloud storage may collectdata for long-term storage. Cloud storage may collect data fromexecution storage. Execution storage may incrementally transmit data tocloud storage. Data within cloud storage may enable developmentimprovement and model training within a development module. Cloudstorage may be accessible by a development module.

The system may include one or more features and analytics modules. Oneor more features and analytics modules may provide for one or more useraccessible features in an application, obtaining data insights fromcollected data of a single user or a number of users, or both. One ormore features and analytics modules may include a user features andabilities module, data analytics module, execution data managementmodule, or a combination thereof. A user features and abilities modulemay function to host an application accessible by one or more personalcomputing devices. A user features and abilities module may allow foruser to see their own data (e.g., video streams), trends, detectedhealth events, behaviors, the like, or any combination thereof. A userfeatures and abilities module may allow for a user to see data specificto similar groups, users as a whole, users in certain demographics, andthe like. User features and abilities module may access data analytics.Data analytics may execute execution data management. Execution datamanagement may utilize machine learning, artificial intelligence, orboth. Execution data management may utilize supervised and/orunsupervised learning. Execution data management may utilize one or moremachine learning models. One or more machine learning models may includesupervised learning models including linear regression, logisticregression, support vector machine, the like, or a combination thereof.One or more machine learning models may include unsupervised learningmodels including hierarchal clustering, k-means, self-organizing map,the like, or a combination thereof. One or more machine learning modelsmay allow for effective analysis of user data, extraction of key datatrends and insights, or both. Analytics may then be used by the userfeatures and abilities module. The user features and abilities modulemay allow for trend predictions and data interpretation to provide aconcise and comprehensive overview of one or more users' activity. Userfeatures and abilities module may provide for real-time alerts, pushnotifications, and/or displays on user interfaces via one or morepersonal computing devices.

The system may include a development module. The development module mayfunction to collect and organize data received by the system, providecontinuous development improvement and model training, or both. Thedevelopment module may include a development data management module,data analytics module, development testing module, model and featuredevelopment module, the like, or a combination thereof.

The development data management module may collect and organize data fortraining. The development data module is in communication with one ormore storage device modules. The development data module collects andorganizes data from cloud storage. The organized data may be referred toas one or more training data sets. One or more training datasets mayalso be obtained from publicly or privately available datasets which arestored within cloud storage. The development data module enablesimprovement and training opportunities for current and new models. Themodels may be located in the model and feature development module. Thedevelopment data management module may store the collected and organizeddata within one or more databases. The development data managementmodule may store the collected and organized data within a developmentcloud training database.

The data analytics module may provide for data insights which can allowfor creation and optimization of one or more current or new machinelearning models, artificial intelligence models, or both. The dataanalytics module may include one or more sub-modules. One or moresub-modules may include a visualization module, data analyticsgeneration module, feature extraction module, the like, or a combinationthereof. A visualization model may be what enables visualization of thedata via one or more user interfaces. A data analytics generation modulemay provide for detailed analytics. The data analytics generation modulemay use data from one or more databases, such as a development cloudtraining database. The data analytics generation module may use machinelearning models to analyze the data. The one or more machine learningmodels may be trained via supervised training models, unsupervisedtraining models, or both. Supervised training models may include linearregression, logistic regression, support vector machine, the like, orany combination thereof. Unsupervised training models may includehierarchical clustering, k-means, and the self-organizing map, the like,or a combination thereof. The resulting data insights from the dataanalytics model may enable a feature extraction model. A featureextraction model may allow for specific features to be isolated andextracted from the resulting data. Isolated and extracted data andfeatures may be saved within one or more databases, such as an extractedfeatures database. The development data management module, dataanalytics module, or both may break down data for use of training theone or more machine learning models.

Data types to be used for training may include one or more videostreams, still images, frames, sound, the like, or a combinationthereof. Video streams may be received in one or more video codingformats. One or more video coding formats may include H.264, HEVC, VP8,VP9, the like, or a combination thereof. Video frames, images, or bothmay be received in one or more image formats. One or more image formatsmay include .png, .jpeg, .gif, the like, or any combination thereof.Data information in addition to or appended to one or more videostreams, still images, frames, sound or the like (e.g., data labels) mayinclude one or more data arrays. Data arrays may include integers,floats, decimals, pixels, NumPy values, text, the like, or a combinationthereof. Data labels may be provided from data or manually labeled in a.csv, .txt, or similar file format. For video stream training and/ortesting, frames may be obtained at 5 frames per second or greater, about10 frames per second or greater, about 15 frames per second or greater,or even about 20 frames per second or greater. The video stream may bebroken down to about 90 frames per second or less, about 60 frames persecond or less, or even about 30 frames per second or less.

The model and feature development module may provide a method forimproving and training new and current models. The new and existingmodels may be machine learning and/or artificial intelligence models.The development testing module may acquire and partition data from oneor more databases. The one or more databases may be a development cloudtraining database, extracted feature database, or both. Partitioneddatasets may be labeled. Labeling may be manual, automatic, or both.Labeling may include a format, type of data, associated health conditionand/or health event to a record, demographic information related tousers within the data, or any combination thereof. Once labeled, datamay be stored within a training database. Model training may occur usingthe data from the training database. Model training may occur by firstacquiring the data from the training database. After acquiring the data,the data may go through a preprocessing step. Preprocessing may bedependent on the type of training model. Deep learning and supervisedlearning algorithms including but are not limited to linear regression,logistic regression, and Support Vector Machine are used in order totake advantage and utilize all of the potentially available data typeswithin training database. Deep learning or deep neural networks (DNNs)are widely accepted and used by data scientists and engineers toaccurately classify the different parts of video frames. DNNpreprocessing may involve converting video streams to individual framesand/or acquiring frames within an extraction database. Next, the faceswithin the frames may be extracted and the extractions' sizes may bestandardized for input into the model. The data may be segmented intotraining and validation datasets with corresponding labels and then arefed through a series of convolution layers with relu activation, poolinglayers, and connected to a flatten layer and a fully connected (dense)layer with a SoftMax classifier. DNN model structure can change or bemodified in order to improve model performance, including but notlimited to adding or subtracting convolution layers, altering the inputnode sizes, changing the activation functions and/or hyperparameteroptimization. Supervised learning may be useful for training on textand/or numerical values. Supervised learning preprocessing involvesorganization of the training data into rows of input data matched withthe corresponding label. Then, feeding the data into one of severalsupervised learning models, including but not limited to linearregression, logistic regression, and Support Vector Machines. Afterpreprocessing, the model begins to be trained. Model training may takeseveral hours to days for deep neural networks and/or several hours forsupervised learning models. Once the model is trained, the model may beevaluated by testing individual frames and/or data rows and verifyingthe performance. The model and feature development module may providefor a useable system with tested models. Once a sufficient model hasbeen trained and evaluated, the model may be configured for test use,saved in a model database, or both. Poor model performance causesprediction and detection inaccuracies, which integers with the mainintention for this invention.

The development testing module may provide for rigorous testing of newand existing models. The development testing module is substantiallysimilar to the model and feature development module. The developmenttesting module includes partitioning one or more training datasets,labeling data, storing data within a testing database, acquiring thedata from the testing database, data preprocessing, training the model,evaluating the results, the like, or a combination thereof.

The system may include one or more modules. The one or more modules mayrefer to hardware, software, or both. Modules may be physical componentswithin the system. Modules may be processes executable by one or moreprocessors and stored within one or more memory storage devices.

Facial Recognition and Monitoring Method (“FRM Method”)

The present teachings may provide for a facial recognition andmonitoring method (also referred to as “FRM method”). The FRM method maybe accessible by, stored within, and/or executed by a recognitiondevice, personal computing device, user interface, cloud computing anddata management module, the like, or a combination thereof. The FRMmethod may be software stored in one or more application layers, cloudstorage computing storage, applications of a personal computing device,the like, or a combination thereof. At least a portion of the FRM methodmay reside outside of the recognition device, be accessible by therecognition device, be located within a cloud-based network, be locatedwithin a cloud computing and data management module, or a combinationthereof. The entire FRM method may be permanently or temporarily storedwithin the recognition device, cloud-based network, cloud computing anddata management module, the like, or a combination thereof. The FRMmethod may be executed by an image processing unit of a recognitiondevice, one or more processors of one or more cloud computing modules,the like, or a combination thereof. The FRM method may be particularlyuseful in identifying one or more users via facial recognition,monitoring the one or more users for the presence of one or more healthevents, detecting the presence of one or more health events, the like,or a combination thereof. The FRM method may be particularly useful inidentifying one or more drivers and/or passengers of a vehicle,detecting the presence of one or more health events of a driver and/orpassenger via facial recognition, or both. The FRM method may include aplurality of steps. The FRM method may be automatically executed by therecognition device, one or more image processing units, or both. The FRMmethod may include one or more of the following steps: videoacquisition, preprocessing, face extraction, facial recognition, eventgeneration, the like, or any combination thereof.

The FRM method may be automatically triggered by one or more camerastream inputs, video stream inputs, or both. One or more cameras mayhave one or more users come into the line of sight of the camera. Uponcoming into the line of sight of the camera, a video stream may becommenced. The video stream may be transmitted from the one or morecamera modules to one or more image processing units. The one or moreimage processing units upon receiving a video stream may initiate videoacquisition. One or more video streams may also be provided by one ormore personal computing devices, web servers and/or browsers, one ormore video stream management networks, the like, or a combinationthereof. While a recognition device may provide for an integrated andeasy-to-use device and system, the ability to process an incoming videostream input from other devices provides flexibility for the FRM systemand method.

The FRM method may include video acquisition. Video acquisition mayfunction to collect one or more outputs of one or more sensing devices.Video acquisition may function to collect one or more outputs of one ormore cameras. Video acquisition may function to collect one or moreimages, videos, frames, sounds, the like, or any combination thereof.Video acquisition may function to store one or more incoming videostreams for further analysis. Video acquisition may be automaticallyexecuted by a recognition device, image processing unit, cloud computingmodule, cloud computing and data management module, the like, acombination thereof. Video acquisition may be automatically executed byone or more processors of an image processing unit, cloud computingmodule, or both. Video acquisition may be automatically executed upon acamera stream input being detected, one or more video streams beingreceived by an image processing unit or cloud computing module, orcombination thereof. Commencement of video acquisition may trigger arecording service. A recording service may be a process for recordingthe incoming video stream. A recording service may be a process whichincludes transferring and storing the incoming video stream within therecognition device, image processing unit, storage medium, remotestorage device, cloud network, a device and data management module, thelike, or any combination thereof. The incoming video stream may bereceived from the camera by the image processing unit and transmittedfor storage in the storage medium of the image processing unit. Theincoming video stream may be received from the camera by the imageprocessing unit and transmitted for storage in a remote storage device,a device and data management module, or both. The incoming video streammay be transmitted for storage by the image processing unit toward thestorage medium, remote storage device, device and data management modulein any sequence, simultaneously, or a combination thereof. The imageprocessing unit may associate the incoming video stream with one or moreidentifiers prior to storage. The one or more identifiers may includeone or more users, recognition devices, timestamps, geolocations, thelike, or any combination thereof. After video acquisition, the FRMmethod moves on to preprocessing.

The FRM method may include preprocessing. Preprocessing may function tobreakdown the incoming video stream into a format which can be furtheranalyzed by the image processing unit. Preprocessing may access anincoming video stream from one or more storage mediums, remote storagedevices, device and data management modules, the like, or a combinationthereof. Preprocessing may be automatically executed by the one or moreprocessors of the image processing unit, cloud computing module, cloudcomputing and data management module, the like, a combination thereof.Preprocessing may function to break down the incoming video stream intoone or more frames. The video stream may be broken down to a frame rateabout equal to or less than a frame rate captured by the camera. Thevideo stream may be broken down to about 5 frames per second or greater,about 10 frames per second or greater, about 15 frames per second orgreater, or even about 20 frames per second or greater. The video streammay be broken down to about 90 frames per second or less, about 60frames per second or less, or even about 30 frames per second or less.The one or more frames may be used by one or more subsequent stepsand/or sub-steps of the FRM method. Preprocessing may include initiatingmodel identification, environmental analysis, or both.

The FRM method may include model identification. Model identificationmay be part of preprocessing. Model identification may function toidentify one or more models suitable for face extraction, facialrecognition, or both. Model identification may function to identify oneor more optimal models for face extraction, facial recognition, or both.During model identification, the image processing unit, cloud computingand data management module, or both may evaluate one or more frames.Based on data from the one or more frames, model identification maydetermine which models may be used. During model identification, theimage processing unit, processor of the recognition device, cloudcomputing module, cloud computing and data management module, the like,a combination thereof may access and extract one or more models from aremote storage device, cloud-based network, or both; transfer the one ormore models to a memory medium; or both. The one remote storage device,cloud network, or both may be accessible via one or more networkconnections of the recognition device, wired and/or wireless connectionsof the cloud-based network, or both. The one or more models may then beutilized by the processor of the image processing unit, cloud computingmodule, cloud computing and data management module, the like, acombination thereof to further the FRM method.

The FRM method may include environmental analysis. Environmentalanalysis may be part of preprocessing. Environmental analysis mayfunction to reduce and/or remove one or more environmental influencesfrom one or more frames. Environmental influences may include differentlighting levels (e.g., illumination), multiple individuals captures inthe video stream and associated individual frames, background locatedbehind the individual, pose, scale, distance of the user to the camera,gestures of the user, the like, or any combination thereof.Environmental analysis may be completed before, simultaneous with, orafter model identification. Environmental analysis may be completed byan image processing unit of the recognition device, cloud computingmodule, cloud computing and data management module, the like, acombination thereof. Environmental analysis may be completed by one ormore processors of the recognition device, cloud computing module, cloudcomputing and data management module, the like, or a combinationthereof. After preprocessing, the FRM method may move on to faceextraction.

The FRM method may include face extraction. Face extraction may functionto locate and extract one or more faces and facial data from one or moreframes. Face extraction may utilize one or more models identified bymodel identification. Face extraction may utilize one or more framesafter environmental analysis. Face extraction may be completed by animage processing unit of the recognition device. Face extraction may beautomatically executed by one or more processors of a recognitiondevice, cloud computing module, cloud computing and data managementmodule, the like, a combination thereof. Face extraction may beautomatically executed after preprocessing such that one or moreindividual frames are available for analysis. Face extraction finds oneor more faces within a frame, crops one or more pixels of one or morefaces, determines a pose of one or more users, determines and extractsone or more features and/or measurements of one or more faces, or anycombination thereof. Face extraction may include face detection, posedetection, facial analysis, the like, or a combination thereof. Facedetection, pose detection, and/or facial analysis may be completedsimultaneously, in sequence, or any variation. Simultaneous evaluationmay allow for face extraction to be completed in a faster time frame.Face detection may allow for a face to first be identified beforeevaluating for further features.

The FRM method may include face detection. Face detection may be part offace extraction. Face detection may function to find one or more faceswithin one or more frames. Once identified, the one or more detectedfaces may have their resolution broken down. Face detection may crop oneor more frames upon detection of one or more faces. Face detection maycrop a frame such that the image is focused on one or more specificfaces, is cropped to a predetermined resolution, is cropped to apredetermined shape, the like, or any combination thereof. Cropping mayinvolve cropping the frame such that the image is focused on the faceand background is removed. Cropping may involve cropping the image to astandard shape. A standard shape may be rectangular, square, circular,ovular, the like, or any combination thereof. Cropping may involvecropping the image to a predetermined resolution. An image may becropped to a resolution equal to or less than capable of capturing by asensing device (e.g., camera). An image may be cropped to be about 150pixels or greater, about 200 pixels or greater, or even about 300 pixelsor greater in one or more directions (e.g., height and/or width). Animage may be cropped to be about 1080 pixels or less, about 720 pixelsor less, or even about 640 pixels or less in one or more directions(e.g., height and/or width). Cropping may allow for one or more framesto be evaluated with a consistent size and resolution. Cropping mayallow for one or more models to identify and classify the face by one ormore models. One or more exemplary models may include deep neuralnetwork, cascade classifier algorithm, the like, or a combinationthereof.

The FRM method may include pose detection. Pose detection may be part offace extraction. Pose detection may function to determine a position andorientation of one or more faces in one or more frames. Pose detectionmay calculate one or more degrees, two or more, or three degrees offreedom of a user's head pose. The degrees of freedom may include yaw,pitch, and roll. Pose detection may allow for the FRM method todetermine where a user is looking (e.g., the road, a fellow passenger,down and away from the road). Pose detection may allow for the FRMmethod to determine motion (e.g., change in position) of a user's faceover a sequence of frames; compensate for a user's facial positionduring facial recognition and/or event generation; or both.

The FRM method may include facial analysis. Facial analysis may be partof face extraction. Facial analysis may function to extract one or morefeatures of a user's face from one or more frames. One or more featuresmay include a color, location, measurements, status, or a combinationthereof of one or more facial features. One or more facial features mayinclude eyes, ears, nose, mouth, and the like. A status may includeopen, closed, partially closed, looking away, the like, or a combinationthereof. Location may include slant of eyes, up or downtown of lips,arch of eyebrows, the like, or a combination thereof. One ormeasurements may include eye distance (eye center to eye center), nosewidth and/or height, lip width and/or height, eye width and/or height,ear distance, ear width and/or height, the like, or a combinationthereof. One or more locations and/or measurements of one or morefeatures may allow for one or more changes to be determined over asequence of frames, a baseline to be determined for a user, or acombination thereof.

The FRM method may include facial recognition. Facial recognition mayfunction to classify one or more faces from one or more frames.Classification may allow for specific features, such as those extractedfrom facial extracted, to be classified, given a status, or both. Astatus may be a health event or a regular event. Facial recognition mayutilize one or more models identified by model identification. Facialrecognition may utilize the extracted faces from face extraction. Facialrecognition may be automatically executed by one or more processors of arecognition device, cloud computing module, cloud computing and datamanagement module, the like, a combination thereof. Facial recognitionmay be automatically executed by an image processing unit, cloudcomputing module, cloud computing and data management module, the like,or a combination thereof. Facial recognition may be automaticallyexecuted after one or more faces and/or facial data are extracted withface extraction. Facial recognition may use one or more arrays, faciallandmarks, user data history, frame data, or any combination thereof.Facial recognition may determine one or more statuses of one or morefaces, track facial changes of one or more users over a sequence offrames, correct inaccurate detections of a user based on systemcustomization, or any combination thereof. Facial recognition mayinclude full face recognition, partial face recognition, behaviorrecognition, the like, or a combination thereof.

The FRM method may include full face recognition. Full face recognitionmay be part of facial recognition. Full face recognition may function toclassify a status of one or more features of a face from one or moreframes. Full face recognition may include breaking one or more extractedfaces from one or more frames into one or more pixel arrays. The one ormore pixel arrays may represent one or more orthogonal components offace encoding. The one or more pixel arrays may be used by one or moremodels. One or more models may include deep neural networks,convolutional neural networks, the like, or any combination thereof.Using the one or more models, the one or more processors of therecognition device determine a status of one or more features of a face.

The FRM method may include partial face recognition. Partial facerecognition may be part of facial recognition. Partial face recognitionmay function to locate one or more facial landmarks, perform measurementchange calculations of one or more facial features, determine changes ofone or more facial features, or any combination thereof on one or moreextracted faces from one or more frames. Partial face recognition mayutilize one or more models. The one or more models may accessible frommodel identification, stored within the cloud network, available fromthird party frameworks and accessible via the cloud network, the like,or a combination thereof. One or more models may include Dlib, as athird-party framework. Partial face recognition may locate one or morefacial landmarks within a face. Facial landmarks may include the innerand outer locations one or more eyebrows; the outer locations of a nasalbridge; the inner, outer, top, and bottom locations of one or more eyes;the centers of one or more pupils; the color of one or more irises; thelocation of the top of the nose; the outer locations of the top andbottom portions of both left and right nose alars, the outer corners ofthe mouth; the center of the upper lip, the center of the bottom lip;the bottom of the upper lip; the outer edges of ears; the top and bottomof left and right ears; the like, or a combination thereof. Partial facerecognition may determine one or more measurements of and/or between oneor more facial landmarks. One or more measurements may from one or moreframes may be compared with one or more measurements from one or moreother frames. Partial recognition may allow for physical changes in oneor more measurements to be identified over a sequence of frames from avideo stream.

The FRM method may include behavior recognition. Behavior recognitionmay be part of facial recognition. Behavior recognition may function tocustomize data related to a user, provide a baseline of data for facialdata of a user, correct inaccurate detections relate to facial data of auser, or any combination thereof. Behavior recognition may allow fordetermination of emotions, gender, the presence and/or absence ofmakeup, hairstyle and/or color differences, age progression, the like,or any combination thereof of one or more users. For example, behaviorrecognition may adapt to recognize a user is wearing no makeup or moremakeup than usual. Behavior recognition may recognize that a user isexcited, happy, sad, angry, or the like. Behavior recognition mayrecognize a user is aging over time. Behavior recognition may recognizea hairstyle and/or color difference of a user. Based on behaviorrecognition, face recognition and partial recognition can be furtherupdated before event generation.

The FRM method may include event generation. Event generation functionsto generation frame analysis results from the FRM method. Eventgeneration may transmit one or more frame analysis results to a storagemedium, remote storage device, device and data management module, thelike, or a combination thereof. Event generation may utilize one or moremodels identified by model identification. Event generation may utilizeone or more outputs from preprocessing, face extraction, facialrecognition, or a combination thereof. Event generation may beautomatically executed by one or more processors of a recognitiondevice, cloud computing module, cloud computing and data managementmodule, the like, or a combination thereof. Event generation may beautomatically executed by an image processing unit, cloud computingmodule, cloud computing and data management module, the like, or acombination thereof. Event generation may be automatically executedafter facial recognition determines one or more classifications, status,facial landmarks, user behaviors, or any combination thereof. Eventgeneration may analyze one or more frames. Event generation may appendcollected image data to one or more frames, frame sequences, videosequences, or a combination thereof to create frame analysis results.Collected image data may include labeling a frame, labeling a videosequence, labeling a frame sequence, or any combination thereof.Collected image data may include extracting collected data into one ormore tables, data frames formats, other applicable formats, the like, orany combination thereof. Analyzed frame results may be analyzed by adecision algorithm to generation an interpretation of the frame analysisresults. The interpretation may be the presence and/or absence or anirregular health event, regular event, or both. The interpretation maybe identification of the irregular health event, regular event, or both.

The FRM method may include generating one or more notifications. One ormore notifications may function to alert one or more predeterminedindividuals of a user undergoing an irregular health event, regularevent, or both. One or more notifications may include one or more pushnotifications, text messages, phone calls (e.g., automated and/orprerecorded), SMS notifications, the like, or a combination thereof. Oneor more notifications may be referred to as one or more alerts. One ormore predetermined individuals may be on or more individuals selected bya user. A user may store one or more predetermined individuals into theFRM system via one or more applications, personal computing devices, orboth. Data saved with respect to a predetermined individual may includea phone number, name, email, the, like, or a combination thereof. One ormore notifications may be sent from a personal computing device, thesystem, physical data center, application, over a network, the like, orany combination thereof. One or more notifications may be initiated byone or more processors, image processing units, cloud computing and datamanagement modules, the like, or a combination thereof.

The FRM method may include identifying and activating one or moreresponse protocols. One or more response protocols may aid in placingthe user in a position of safety, removing and/or reducing the risk ofharm from other individuals in proximity to the user, or both. One ormore response protocols may include notifying one or more emergencyservices, providing a geolocation of a user, engaging one or more safetyprotocols of a vehicle, or a combination thereof. One or more safetyprotocols of a vehicle may include engaging driver assistance technologyto aid in maneuvering the vehicle to safety. Safety of the vehicle mayinclude slowing down, braking, stopping, initiating lane control,controlling the acceleration, moving the vehicle to the side of theroad, a nearby parking spot (e.g., on-street parking, parking lot), thenearest emergency services location, the like, or a combination thereof.By engaging driver assistance technology, the vehicle may be able to berelatively completely maneuvered without the aid of a human driver.Safety protocols of a vehicle may further include turning on one or morehazard lights, sounding a horn or alarm, the like, or a combinationthereof.

Vehicle Integration with Driver Assistance

The recognition device and/or system of the present teachings may beable to be integrated with a vehicle. Integration may allow for anadvanced driver assistance system (ADAS) to initiate one or more safetyprotocols of the vehicle, initiate driver assistance technology of avehicle, or both. One or more safety protocols of the vehicle mayinclude turning on hazard lights, sounding a horn, sounding an alarm,vibrating a driver's seat, maneuvering the vehicle to safety withoutassistance from a driver, the like, or any combination thereof.

The recognition device may be in wired or wireless communication with acontroller area network (CAN) bus, controller area network controller,or both. Communication with the CAN allows for the recognition device tocommunicate with the vehicle's electronic control units (ECU). Therecognition device may have one or more wired connections. A wiredconnection may connect with a vehicle's diagnostic link connector (DLC).The diagnostic link connector may also be known as a vehicle's OBD-IIconnector. The OBD-II may provide direct access to the vehicle'sController Area Network (CAN) Bus. CAN provides a simple protocol whichallows for a vehicle's electronic control units (ECUs) to communicate.Via the CAN Bus, the recognition device is able to communication withthe vehicle's ECU.

The recognition device may also use one or more application layers. Theone or application layers may allow for the ADAS software to beconnected to a CAN controller of the vehicle. The application layer maybe provided to a socket layer. The socket layer may connect to asocketCAN. SocketCAN may allow for the custom ADAS software to integrateone or more protocols and kernels alongside a character device driver.One or more protocols may include Protocol Family CAN, protocol familyinternet, or both. A character device driver may be a default CANprotocol and/or kernel of the vehicle. CAN kernels may be used formanaging CAN messages communicated on a CAN bus.

ADAS may use one or more hardware or software standards compatible withone or more programming languages. The hardware or software standardsmay be OpenXC. The one or more programming languages may include Python.The one or more programming languages may translate proprietary CANprotocols into an easy to read format, translate code into a formatunderstanding by the CAN protocols, or both. The one or more hardware orsoftware standards may allow access to any CAN packet via a typical API(application programming interface).

To initiate ADAS and one or more safety protocols of a vehicle, therecognition device may use its wired connection to the vehicle's DLC tocommunicate with the vehicle's CAN. The ADAS may use SocketCAN by way ofprogramming language and hardware or software standards to integrate oneor more custom protocols and/or kernels with the vehicle's CAN. Theprogramming language may by Python. The hardware or software standardsmay be OpenXC scripts. ADAS may send CAN protocol packets to the CAN.The CAN protocol packets may function as the vehicle's method ofcommunication. Vial the protocol packets, the ADAS and recognitiondevice may be able to obtain at least partial control of the vehicle andexecute one or more safety protocols.

ILLUSTRATIVE EXAMPLES

FIG. 1 illustrates the recognition device 10 integrated into a vehicle12. The recognition device 10 includes a camera 14. The camera 14 isintegrated into a rear-view mirror 16. The camera 14 has a line of sight18 on a driver 20.

FIG. 2A illustrates a recognition device 10. The recognition device 10includes a rear-view mirror 16. The rear-review mirror 16 includes acamera 14. The camera 14 is integrated into a housing 22 of therear-view mirror 16.

FIG. 2B illustrates recognition device 10. The system 10 includes acamera 14. The camera 14 is able to be mounted in proximity to arear-view mirror 16. The camera 14 can be mounted onto a windshield,headliner, headliner console, and the like to be in proximity of therear-view mirror 16.

FIG. 3 illustrates a recognition device 10. The recognition device 10includes a housing 22. The recognition device 10 includes a camera(e.g., camera module) 14. As an alternative, the camera 14 may beoutside of and separate from the housing 22. The camera 14 receives acamera input stream 35. The camera input stream 35 occurs when a user 84is in view of the camera 14. The camera module 14 is connected to and incommunication with an image processing unit 34. The recognition device10 includes a processor 26, memory medium 28, storage medium 30, andgraphics unit processor 32. The image processing unit 34 may include acircuit (e.g., circuit board) 33. The circuit 33 supports and connectsthe processor 26, memory medium 28, storage medium 30, and graphics unitprocessor 32. The image processing unit 34 is in electricalcommunication a power supply 38. The power supply 38 is connected to apower input 39. The power input 39 is connected to a power source 40.The recognition device 10 includes a network connection 42. The networkconnection 42 may be an IoT connection. The network connection 42 allowsfor the recognition device 10 to be in communication with a remotestorage device 48. The remote storage device 48 is in communication withand part of a network 50. The network 50 also includes a cloud-basednetwork 54. The recognition device 10 includes an application layer 46.The application layer 46 may be part of or accessible by the imageprocessing unit 34. The application layer 46 may include one or morefacial recognition and monitoring instructions stored therein which areaccessible for execution by the image processing unit 34. Therecognition device 10 includes is connected to an advanced driverassistance system 49.

FIG. 4 illustrates a method for facial recognition and monitoring 122 ofa recognition device 10 (not shown). The method 122 may be part of orstored within an application layer 46. For the method 122 to commence, acamera input stream 35 is received by a camera 14 (not shown). Thecamera input stream 35 provides the video stream input for the method122 to commence. Once the camera input stream 35 is received a camera 14(not shown) and an image processing unit 34, a step of video acquisition130 takes the video stream input and initiates a recording service 128.The image processing unit 34 executes the recording service 128. Therecording service 128 records the video stream input such that therecognition device 10 stores the recorded video stream input in astorage medium 30 (not shown) of the recognition device 10. The recordedvideo stream input is able to be transferred from the storage medium 30to the remote storage device 48 (not shown) via a network connection 42(not shown). The recorded video stream may be able to be transferredfrom the remote storage device 48 or the storage medium 30 to acloud-based network 54, such as into a device and data management module64.

After video acquisition 130, the method moves on to preprocessing 132.During preprocessing 132, the image processing unit 34 passes one ormore individual frames from the video stream to a model identificationprocess 136. During the model identification process 136, the imageprocessing unit 34 may decide one or more optimal models to use on theframe for facial extraction and recognition. During the modelidentification process 136, the image processing unit 34 may load one ormore optimal models from a remote storage device 48 to the imageprocessing unit 34, such as into a memory medium 28 (not shown). After,before, or simultaneous with the model identification process 136, anenvironmental analysis process 138 is initiated. During theenvironmental analysis 138, the image processing unit 34 reduces and orremoves environmental influence from one or more frames of the videostream.

After preprocessing 132, the method moves on to face extraction 140. Theone or more individual frames from preprocessing 132 are used for faceextraction 140. During face extraction 140, the image processing unit 34(not shown) detects and extracts faces and/or facial data from one ormore frames. The step of face extraction 140 includes sub-steps facedetection 141, pose detection 142, and facial analysis 143. During facedetection 141, the image processing unit finds one or more faces withinthe one or more frames. During face detection 141, the image processingmay crop one or more pixels making up one or more faces of one or moreusers. Finding and cropping may allow for one or more deep neuralnetworks to classify the one or more faces. During pose detection 142,the image processing unit may determine bodily position of the one ormore users from the one or more frames. Bodily position may include thethree degrees of freedom of a user's head, yaw, pitch, and/or roll andrepresent a position and orientation of a user's face. During facialanalysis 143, the image processing unit may extract one or more specificfeatures and/or measurements exhibited by a face of a user within aframe.

After face extraction 140, the method moves on to facial recognition148. During facial recognition 148, the image processing unit 34 (notshown) uses one or more pre-trained models to classify the one or morefaces extracted from the one or more frames. Facial recognition 148includes sub-steps full face recognition 144, partial face recognition145, and behavior recognition 146. For full face recognition 144, theimage processing unit uses the pixel arrays of the extracted faces fromthe one or more frames into one or more learning models, which classifythe specific features within the face. For partial face recognition 144,the image processing unit uses one or more learning models to locatefacial landmarks within the face, performing one or more measurements ofthe face, and determine one or more changes of the face throughout asequence of individual frames. For behavior recognition 145, the imageprocessing unit accesses a user's data history and data from one or moreindividual frames, corrects inaccurate detections of a user, provides alevel of customization relative to the user.

After facial recognition 148, the method moves to alert generation 154.During alert generation 143, the image processing unit generates one ormore analysis results from facial recognition 148. The image processingunit transmits the one or more analysis results to the remote storagedevice 48, the device and data management module 64, or both. The imageprocessing unit associated analysis results from each individual frameto its specific frame. The analysis results for one or more frames areevaluated by the image processing unit and compiled by a decisionalgorithm. The decision algorithm determines if a health event ispresent in a user. Analysis results may also be transmitted to arecognition device event data manager 68.

FIG. 5 illustrates a schematic (e.g., architecture) of a facialrecognition and monitoring system 50 (e.g., system). The system 50 isconfigured as an overall network. The system 50 includes a physical datacenter 52. The physical data center 52 hosts a cloud-based network 54.The cloud-based network 54 includes a network architecture 56. Thenetwork architecture 56 includes a development module 58, data storagemodule 60, features and analytics module 62, device and data managementmodule 64, and cloud computing and data management module 66. Thecloud-based network 54 is in communication with a user-interface 69.

The cloud-based network 54 is in communication with a recognition devicemanagement network 70. The recognition device management network 70 isin communication with the device and data management module 64. Therecognition device management network 70 includes one or more remotestorage devices 48. The one or more remote storage devices 48 mayinclude one or more main storage devices 48 a and one or more nodestorage devices 48 b. One or more recognition devices 10 may be incommunication with one or more remote storage devices 48. One or morerecognition devices 10 may be in communication with a node storagedevice 48 b.

The cloud-based network 54 is in communication with a video streammanagement network 74. The video stream management network 74 is incommunication with a cloud computing and data management module 66. Thevideo stream management module 74 includes one or more video streamclouds 76. The video stream cloud 76 may be in communication with a uservideo cloud 78 and a user data center 80.

The recognition device management network 70 and video stream managementnetwork 74 may receive a camera stream input 35 from one or more users84. One or more users 84 may provide a camera stream input 35 into oneor more recognition devices 10, user video clouds 78, and/or user datacenter 80.

The one or more users 84 are illustrated as exemplary use scenarios. Theone or more users 84 may include one or more drivers and/or passengersof a private passenger vehicle 86; one or more drivers of publictransportation 88; one or more individuals in any setting 90 such as viatheir mobile device; and one or more passengers in public transportation90.

FIG. 6 illustrates the network architecture 56 of the cloud-basednetwork 54. The cloud-based network 54 includes a development module 58,data storage module 60, features and analytics module 62, device anddata management module 64, and cloud computing and data managementmodule 66. The device and data management module 64 includes a deviceevent management sub-module 68, device management sub-module 70, anddevice storage sub-module 72. The cloud computing and data managementmodule 66 includes a cloud event management sub-module 74, cloudcomputing sub-module 76, and cloud computing storage sub-module 78.

Data storage module 60 includes cloud storage 80, execution eventstorage 82, and execution storage 84. The features and analytics module62 includes a data analytics module 86 and user features and abilitiesmodule 88. The development module 58 includes a model and featuredevelopment module 90, data analytics module 92, development datamanagement module 94, and development testing module 96.

FIG. 7 illustrates a development module 58. The development module 58enables continuous development improvement and model training for thesystem 50. The development module 58 includes a model and featuredevelopment module 90, data analytics module 92, development datamanagement module 94, and development testing module 96.

Development data management module 94 creates datasets from data storedwithin cloud storage 60, 80. Development data management accesses cloudstorage at step 200. After accessing cloud storage, imports collecteddata at step 252. After importing, organizes the data for specificfeature models to be trained at step 204. After organizing, saves theorganized data into a development cloud training database 206.Development data management 240 can also create training datasets frompublicly or privately available datasets uploaded to cloud storage 102.

After creation of the database 206, development module 94 transmits thecollected and organized datasets to a data analytics module 92. Thedatasets may be data insights used for developing feature models throughthe data analytics module 92. The data analytics module 92 enables datavisualization through a visualization module 250. The data analyticsmodule 92 generates detailed analytics through a data analyticsgeneration module 248. The data analytics generation module 248 utilizedobserved data from database 206. The resulting insights from the dataanalytics generation module 248, enable identification of specificfeatures for isolation and extraction by feature extraction module 252.The one or more specific features which are isolated and extracted aretransmitted and stored within an extracted features database 246.

The model and feature development module 90 is a process method fortraining A.I. and M.L. models. First, data from database 206 ispartitioned for model training and validation by collecting applicabletraining data for the specific feature to be learned, in step 221.Features can be identified from database 246, or by other methods, so aslong as training data is sufficient to support model training. Next,datasets are labeled, either manually or by another process, and savedto training database 212. This may be done as a mitigation step in orderto maintain a detailed record of the amount, format, and type of dataused for training. Model training begins by acquiring data from database212, in step 214. Next, data preprocessing occurs to correctly formatdata for the type of model being used, in step 216. Preprocessingdepends entirely on the type of training model as multiple data typesmay be available for training. Deep learning and supervised learningalgorithms including but are not limited to linear regression, logisticregression, and Support Vector Machine are used in order to takeadvantage and utilize all of the potentially available data types withindatabase 212. Following step 216, the model begins training, step 218,which may take several hours to days for DNNs and/or several hours forsupervised learning models. Once the model is trained, it can beevaluated by testing individual frames and/or data rows and verifyingthe performance, at step 220. Model and feature development 90 is animportant consideration for creating a useable system 50. Once asufficient model has been trained and evaluated, the model is configuredfor test use, at step 244, and saved to a model database 242.

A development testing module 96 allows for rigorous testing of the modelperformance in order to optimize it for deployment. The developmenttesting module 96 starts by creating and partitioning the testingdataset, step 222, and labeling the data for performance evaluation,step 224. The dataset for testing is saved into a testing database 226,and then imported for use at step 228. The imported data is preprocessedappropriately for the type of model to be tested, step 230. Thedifference with the model and feature development module is that inpreprocessing, the data does not contain the labels (e.g., knownoutcomes). The model is subsequently fed the testing data, and theoutputted predictions, in the form of frames, images, text, numericalvalues, and/or any other applicable data output, step 232. Thepredictions are evaluated for accuracy, and the model precision, recalland F₁-Score are calculated, step 234. Models that perform well areconfigured for use in facial recognition and monitoring system, in step240, and saved to model pipeline testing database 238. Models approvedfor development testing, are evaluated a facial recognition andmonitoring testing and training method.

FIG. 8 illustrates a remotely executed method for facial recognition andmonitoring 122 (FRM Method). The FRM method 122 may be executed by acloud-based network 54. The method 122 may be part of or stored within acloud computing and data management module 66. For the method 122 tocommence, a video stream 36 is received by a cloud-based network 54. Thevideo stream is received by a cloud computing and data management module66. The video stream 36 may be received by a wireless or wiredconnection gateway. The video stream 36 may be received from one or moreremotely located personal computing devise, web servers, video streammanagement networks, the like, or a combination thereof. The videostream 36 provides the video stream input for the method 122 tocommence. Once the video stream 36 is received by a cloud computing anddata management module 66, such as an event management module 74, a stepof video acquisition 130 takes the video stream input and initiates arecording service 128. The image processing unit 34 executes therecording service 128. The recording service 128 records the videostream input such that the recorded video stream input is stored withina recording database 260. The recording database 260 may be part of thecloud computing and data management module 66. The recorded video streaminput is able to be transferred from the recording database 260 to cloudstorage 80.

After video acquisition 130, the method moves on to preprocessing 132.During preprocessing 132, the image processing unit 34 passes one ormore individual frames from the video stream to a model identificationprocess 136. During the model identification process 136, the cloudcomputing and data management module 66 may decide one or more optimalmodels to use on the frame for facial extraction and recognition. Duringthe model identification process 136, one or more optimal models may beaccessed from an identified models database 262. After, before, orsimultaneous with the model identification process 136, an environmentalanalysis process 138 is initiated. During the environmental analysis138, the module 66 reduces and or removes environmental influence fromone or more frames of the video stream.

After preprocessing 132, the method moves on to face extraction 140. Theone or more individual frames from preprocessing 132 are used for faceextraction 140. During face extraction 140, the module 66 detects andextracts faces and/or facial data from one or more frames. The step offace extraction 140 includes sub-steps face detection 141, posedetection 142, and facial analysis 143. During face detection 141, themodule 66 finds one or more faces within the one or more frames. Duringface detection 141, the module 66 may crop one or more pixels making upone or more faces of one or more users. Finding and cropping may allowfor one or more deep neural networks to classify the one or more faces.During pose detection 142, the module 66 may determine bodily positionof the one or more users from the one or more frames. Bodily positionmay include the three degrees of freedom of a user's head, yaw, pitch,and/or roll and represent a position and orientation of a user's face.During facial analysis 143, the module 66 may extract one or morespecific features and/or measurements exhibited by a face of a userwithin a frame.

After face extraction 140, the method moves on to facial recognition148. During facial recognition 148, the module 66 uses one or morepre-trained models to classify the one or more faces extracted from theone or more frames. Facial recognition 148 includes sub-steps full facerecognition 144, partial face recognition 145, and behavior recognition146. For full face recognition 144, the module 66 uses the pixel arraysof the extracted faces from the one or more frames into one or morelearning models, which classify the specific features within the face.For partial face recognition 144, the module 66 uses one or morelearning models to locate facial landmarks within the face, performingone or more measurements of the face, and determine one or more changesof the face throughout a sequence of individual frames. For behaviorrecognition 145, the module 66 accesses a user's data history and datafrom one or more individual frames, corrects inaccurate detections of auser, provides a level of customization relative to the user.

After facial recognition 148, the method moves to alert generation 154.During alert generation 143, the module 66 generates one or moreanalysis results from facial recognition 148. The module 66 transmitsthe one or more analysis results to a cloud computing database 264. Thecloud computing database 264 may be located within cloud computingstorage 78. The module 66 associates analysis results from eachindividual frame to its specific frame. The analysis results for one ormore frames are evaluated by the module 66 and compiled by a decisionalgorithm. The decision algorithm determines if a health event ispresent in a user. Analysis results may also be transmitted to a userinterface 69 via an event manager 74. One or more steps of the method122 may be executed by cloud computing 76 (such as shown in FIG. 6 ).Cloud computing 76 may include one or more processors. The method 122may be stored within cloud computing storage 78 and accessed by the oneor more processors.

FIG. 9 illustrates a method 122 for facial recognition and monitoringfor presence of a health event. The method 122 starts and waits foruser(s) to come into view for monitoring 302. When method 122 detectsuser(s) in view 304 of applicable camera(s) and/or additional sensor(s)and/or receives a video stream 36 (not shown), users are initialized atstep 306 into the system. Initializing begins by using facialrecognition models to tag each user actively being monitored. Byidentifying the users, the system is able to differentiate betweenusers. The method 122 then begins to actively monitor each user'sphysicality at step 308. As the system monitors each user, the systemalso analyzes each user's actions and behaviors, and extractingapplicable features from video frames at step 310. This extraction isaccomplished by using the FRM method 122 as illustrated in FIGS. 4 and 8, and a network 50 such as described in FIGS. 5 and 6 . After extractionand analysis 310, the method 122 subsequently updates collected data forfuture use at cloud 54. Based on the collected data, the systemgenerates detections and predictions so as to accurately identify one ormore irregular user events (e.g., abnormal health conditions) activelyoccurring, done at step 312. The detections and predictions from step312 are transmitted and updated within the cloud 54. As the cloud 54receives updates, the cloud 54 updates relevant data to a user interfaceat step 330. The method continues to update the user interface asupdated user data, detections, and predictions are collected throughoutthe method 198.

If the method 122 does not detect a health event, the method 122 repeatsa monitoring loop until the one or more users have exited from view. Ifthe method 122 does detect a health event actively occurring or recentlyoccurred in one or more users at step 316, an immediate alert and/orpush notification 318 is transmitted to one or more applicable persons.Simultaneous with sending the alert and/or push notification, thecollected data is transmitted to the cloud 54.

The method 54 continues by classifying the type of health eventoccurring, having occurred, or that will occur and activating anappropriate response protocol configured based on the irregular eventdetected at step 320. If response actions at step 320 resolved thehealth event, the method collects and transmits all data, features,and/or relevant materials to cloud 54, at step 328. If the event is notresolved, the method 122 returns to step 320 for additional responseprotocol actions.

When the irregular event is identified as resolved, the method 122checks the user's activity status at step 324 to determine if monitoringcan continue. If the user's status returns to active reset user's statusand event details at step 326 and begin new monitoring loop at step 318.If not, method 122 returns to step 332 and awaits new user(s) and/orsystem shutdown.

FIG. 10 illustrates an advanced driver assistance system (ADAS) 49integrated with a vehicle. ADAS 49 uses a wired connection 350 to avehicles diagnostic link connector (DLC) 352. The diagnostic linkconnector 352 provides direct access to a vehicle's Controller AreaNetwork (CAN) bus 354.

ADAS 49 uses an application layer 46. The application layer connects thecustomized ADAS software 49 to the CAN controller 356. The connection ismade through a socket connection 358 to SocketCAN 360. SocketCAN 360allows for integrating with protocols 362, 364 and kernels 366 alongsidethe character device driver.

FIG. 11 illustrates a model testing and training method 400. The modeltesting and training method 400 is used before a model may be deployedfor use, test the model in a production-like execution environment, orboth. Module 410 allows a specific model to be configured for modeltesting and training for the FRM method. This is done by removing acurrent model, an/or adding the new model for testing by way of database238. Applicable modules for configuration include model identification136, environmental analysis 138, face detection 141, pose detection 142,facial analysis 143, full face recognition 144, partial face recognition145, behavior recognition 146, alert generation 154 or any combinationthereof. Following test model configuration and setup, testing data, inthe form of a video stream, is obtained from database 2016 and inputtedinto video acquisition 130. Then, preprocessing 132 passes individualframes to model identification 136, which enables test model to be used,and loads said model from database 238. Next, environmental analysis138, as previously described in figures above, is performed to identifyand mitigate environmental influences within the frame.

Face extraction 140 utilizes face detection 141 to find and extract thepixel representations of the user's face within the frame, and to uploadperformance to database 238. Pose detection 142 is used to calculate thethree degrees of freedom of a user's head pose: yaw, pitch, and roll,representing the position and orientation of the user's face, and toupload performance to database 238. And finally, facial analysis 143 isused to extract one or more specific features and/or measurementsexhibited by the user's face in the frame, and to upload performance todatabase 238.

Facial recognition 148 uses its pre-trained models to classify theface(s) extracted from the frame, and to upload performance to database238. Full facial recognition 711 feeds the pixel arrays of the extractedfaces from the frame into machine learning models (e.g., DNN and CNNmodels) to classify the status of specific features within the face, andto upload performance to database 238. Partial recognition 145 usesmodels from third-party frameworks, such as Dlib, to locate the faciallandmarks within the face, and perform several types of distancecalculations to track a user's physical face changes between the framesof the video stream, and to upload performance to database 238. Andfinally, behavior recognition 146 utilizes a user's data history andrelevant frame data to correct inaccurate detections specific for auser, and to upload performance to database 238.

Following module 148, frame analysis results are generated by module154, and stored into database 238, as well as updated to cloud storage206 by module 94 for saving the model performance in order to comparetested models and identify the most optimal configuration.

Unless otherwise stated, any numerical values recited herein include allvalues from the lower value to the upper value in increments of one unitprovided that there is a separation of at least 2 units between anylower value and any higher value. As an example, if it is stated thatthe amount of a component, a property, or a value of a process variablesuch as, for example, temperature, pressure, time and the like is, forexample, from 1 to 90, preferably from 20 to 80, more preferably from 30to 70, it is intended that intermediate range values such as (forexample, 15 to 85, 22 to 68, 43 to 51, 30 to 32 etc.) are within theteachings of this specification. Likewise, individual intermediatevalues are also within the present teachings. For values which are lessthan one, one unit is considered to be 0.0001, 0.001, 0.01 or 0.1 asappropriate. These are only examples of what is specifically intendedand all possible combinations of numerical values between the lowestvalue and the highest value enumerated are to be considered to beexpressly stated in this application in a similar manner.

Unless otherwise stated, all ranges include both endpoints and allnumbers between the endpoints. The use of “about” or “approximately” inconnection with a range applies to both ends of the range. Thus, “about20 to 30” is intended to cover “about 20 to about 30”, inclusive of atleast the specified endpoints.

The terms “generally” or “substantially” to describe angularmeasurements may mean about +/−10° or less, about +/−5° or less, or evenabout +/−1° or less. The terms “generally” or “substantially” todescribe angular measurements may mean about +/−0.01° or greater, about+/−0.1° or greater, or even about +/−0.5° or greater. The terms“generally” or “substantially” to describe linear measurements,percentages, or ratios may mean about +/−10% or less, about +/−5% orless, or even about +/−1% or less. The terms “generally” or“substantially” to describe linear measurements, percentages, or ratiosmay mean about +/−0.01% or greater, about +/−0.1% or greater, or evenabout +/−0.5% or greater.

The disclosures of all articles and references, including patentapplications and publications, are incorporated by reference for allpurposes. The term “consisting essentially of” to describe a combinationshall include the elements, ingredients, components or steps identified,and such other elements ingredients, components or steps that do notmaterially affect the basic and novel characteristics of thecombination. The use of the terms “comprising” or “including” todescribe combinations of elements, ingredients, components or stepsherein also contemplates embodiments that consist essentially of, oreven consist of the elements, ingredients, components or steps. Pluralelements, ingredients, components or steps can be provided by a singleintegrated element, ingredient, component or step. Alternatively, asingle integrated element, ingredient, component or step might bedivided into separate plural elements, ingredients, components or steps.The disclosure of “a” or “one” to describe an element, ingredient,component or step is not intended to foreclose additional elements,ingredients, components or steps.

It is understood that the above description is intended to beillustrative and not restrictive. Many embodiments as well as manyapplications besides the examples provided will be apparent to those ofskill in the art upon reading the above description. The scope of theinvention should, therefore, be determined not with reference to theabove description, but should instead be determined with reference tothe appended claims, along with the full scope of equivalents to whichsuch claims are entitled. The disclosures of all articles andreferences, including patent applications and publications, areincorporated by reference for all purposes. The omission in thefollowing claims of any aspect of subject matter that is disclosedherein is not a disclaimer of such subject matter, nor should it beregarded that the inventors did not consider such subject matter to bepart of the disclosed inventive subject matter.

What is claimed is:
 1. A method for recognizing and monitoring one ormore users for an occurrence of a health event of the one or more userscomprising: a) detecting the one or more users coming into view of oneor more image sensors and receiving video data related to the one ormore users, wherein the one or more image sensors are in communicationwith one or more processors; b) receiving the video data by the one ormore processors from the one or more image sensors related to the one ormore users upon the detecting of the one or more users coming into viewof the one or more image sensors; c) preprocessing the video data by theone or more processors into frame data, wherein the frame data includesone or more single, batches, sequences or a combination thereof offrames; d) extracting facial data by the one or more processors from theframe data to identify the one or more users, wherein the facial dataincludes one or more extracted faces, one or more numeric arrayrepresentations of one or more faces of the one or more users, one ormore measurements and/or one or more predictions of one or more poses ofthe one or more users, or a combination thereof; e) determining apresence, a probability, and/or an absence of the health event in theone or more users by the one or more processors by comparing the facialdata with one or more stored facial data models accessible by the one ormore processors; f) generating one or more notifications based onrecognizing the presence, the probability, or both of the health eventor not generating the one or more notifications based on the absence ofthe health event; wherein the method includes the one or more processorscontinuing to receive the video data while the one or more users are inview of the one or more image sensors, and wherein the health eventincludes one or more irregular health events which include the one ormore users experiencing a seizure, a heart attack, a stroke, fainting,vomiting, or a combination thereof; wherein the one or more store facialdata models includes a plurality of facial data which is pre-storedwithin one or more non-transitory storage mediums and already associatedwith one or more irregular health conditions which are associated withthe one or more irregular health events; and wherein the one or morestored facial data models are determined using supervised machinelearning in which the plurality of facial data is stored and associatedwith the one or more irregular health conditions.
 2. The method of claim1, wherein the video data includes one or more video files, image files,frames, or any combination thereof.
 3. The method of claim 2, whereinthe video data is received by a recording service module and associatedwith one or more identification data; and wherein the one or moreidentification data includes one or more video labels, timestamps,camera identifiers, user identifiers, or any combination thereof.
 4. Themethod of claim 2, wherein the video data is transmitted to one or moreremotely located non-transitory storage mediums.
 5. The method of claim1, wherein extracting the facial data includes utilizing one or moreface detection models, pose detection models, facial analysis models, ora combination thereof.
 6. The method of claim 5, wherein the one or moreface detection models convert facial image data into the one or morenumeric array representations of the one or more faces of the one ormore users.
 7. The method of claim 5, wherein the one or more posedetection models convert facial image data into the one or moremeasurements, predictions, or both related to one or more facial poses,body poses, or both of the one or more users.
 8. The method of claim 1,wherein the one or more stored facial data models are determined by oneor more machine learning networks.
 9. The method of claim 8, wherein theone or more machine learning networks include one or more convolutionalneural networks (CNN), one or more Dlib machine learning algorithms, orany combination thereof.
 10. The method of claim 1, wherein the one ormore notifications notifiers includes one or more hazard lights of avehicle.
 11. The method of claim 1, wherein the one or morenotifications includes one or more calls to one or more emergencyservices.
 12. The method of claim 1, wherein the one or morenotifications includes one or more calls, text notifications, or both toone or more pre-determined individuals.
 13. The method of claim 1,wherein the one or more image sensors are part of one or more cameras.14. The method of claim 1, wherein the one or more image sensors arelocated in a same device as, remote from, or both the one or moreprocessors.
 15. The method of claim 1, wherein upon detecting thepresence and/or the probability of the health event, the method includesidentifying and activating one or more response protocols.
 16. A systemfor facial recognition and monitoring for performing the method of claim1, comprising a recognition device including: i) one or more camerashaving the one or more image sensors; and ii) one or more imageprocessing units in communication with the one or more camerasincluding: a) the one or more processors, b) one or more graphicsprocessors, c) the one or more non-transitory storage mediums, and d)one or more internet connections.
 17. The system of claim 16, whereinthe recognition device is integrated into a vehicle; and wherein the oneor more cameras are configured to have a line of sight on one or moredrivers, one or more passengers, or both within the vehicle.
 18. Themethod of claim 1, wherein the method includes upon recognizing thepresence and/or the probability of the health event, transmitting one ormore emergency signals from the one or more processors to a vehicle toenable one or more safety protocols including initiating a driverassistance technology to control driving of the vehicle in which the oneor more users are located, such that the vehicle drives to and reaches asafe parking destination and/or turns on one or more emergency notifiersof the vehicle.
 19. A method for recognizing and monitoring one or moreusers for an occurrence of a health event of the one or more userscomprising: a) detecting the one or more users coming into view of oneor more image sensors and receiving video data related to the one ormore users, wherein the one or more image sensors are in communicationwith one or more processors; b) receiving the video data by the one ormore processors from the one or more image sensors related to the one ormore users upon the detecting of the one or more users coming into viewof the one or more image sensors; c) preprocessing the video data by theone or more processors into frame data, wherein the frame data includesone or more single, batches, sequences or a combination thereof offrames; d) extracting facial data by the one or more processors from theframe data to identify the one or more users, wherein the facial dataincludes one or more extracted faces, one or more numeric arrayrepresentations of one or more faces of the one or more users, one ormore measurements and/or one or more predictions of one or more poses ofthe one or more users, or a combination thereof; e) determining apresence, a probability, and/or an absence of the health event in theone or more users by the one or more processors by comparing the facialdata with one or more stored facial data models accessible by the one ormore processors; f) generating one or more notifications based onrecognizing the presence, the probability, or both of the health eventor not generating the one or more notifications based on the absence ofthe health event; and g) upon recognizing the presence and/or theprobability of the health event, transmitting the one or more emergencysignals from the one or more processors to a vehicle to enable one ormore safety protocols including initiating a driver assistancetechnology to control driving of the vehicle in which the one or moreusers are located, such that the vehicle drives to and reaches a safeparking destination and/or turns on one or more emergency notifiers ofthe vehicle; and wherein the method includes the one or more processorscontinuing to receive the video data while the one or more users are inview of the one or more image sensors; wherein the health event includesone or more irregular health events which include the one or more usersexperiencing a seizure, a heart attack, a stroke, fainting, vomiting, ora combination thereof; wherein the one or more stored facial data modelsare determined by one or more machine learning networks; wherein the oneor more machine learning networks include one or more convolutionalneural networks (CNN), one or more Dlib machine learning algorithms, orany combination thereof; wherein the one or more stored facial datamodels includes a plurality of facial data which is pre-stored withinone or more non-transitory storage mediums and already associated withone or more irregular health conditions which are associated with theone or more irregular health events; and wherein the one or more storedfacial data models are determined using supervised machine learning inwhich the plurality of facial data is stored and associated with the oneor more irregular health conditions; wherein extracting the facial dataincludes utilizing one or more face detection models, pose detectionmodels, facial analysis models, or a combination thereof; wherein theone or more face detection models convert facial image data into the oneor more numeric array representations of faces of the one or more users;and wherein the one or more pose detection models convert the facialimage data into one or more measurements, predictions, or both relatedto one or more facial poses, body poses, or both of the one or moreusers.
 20. The method of claim 19, wherein the one or more emergencynotifiers includes one or more hazard lights of the vehicle; wherein theone or more notifications includes one or more calls to one or moreemergency services; and wherein the one or more notifications includesone or more calls, text notifications, or both to one or morepre-determined individuals.